Everything about Statistical Significance totally explained
In
statistics, a result is called
statistically significant if it's unlikely to have occurred by
chance. "A statistically significant difference" simply means there's statistical evidence that there's a difference; it doesn't mean the difference is necessarily large, important, or significant in the common meaning of the word.
The
significance level of a test is a traditional
frequentist statistical hypothesis testing concept. In simple cases, it's defined as the probability of making a decision to reject the null hypothesis when the
null hypothesis is actually true (a decision known as a
Type I error, or "false positive determination"). The decision is often made using the
p-value: if the p-value is less than the significance level, then the null hypothesis is rejected. The smaller the p-value, the more significant the result is said to be.
In more complicated, but practically important cases, the significance level of a test is a probability such that the probablility of making a decision to reject the null hypothesis when the
null hypothesis is actually true is
no more than the stated probability. This allows for those applications where the probability of deciding to reject may be much smaller than the significance level for some sets of assumptions encompassed within the null hypothesis.
Use in practice
The significance level is usually represented by the Greek symbol, α (alpha). Popular levels of significance are 5%, 1% and 0.1%. If a
test of significance gives a p-value lower than the α-level, the null hypothesis is rejected. Such results are informally referred to as 'statistically significant'. For example, if someone argues that "there's only one chance in a thousand this could have happened by coincidence," a 0.1% level of statistical significance is being implied. The lower the significance level, the stronger the evidence.
In some situations it's convenient to express the statistical significance as 1 − α. In general, when interpreting a stated significance, one must be careful to note what, precisely, is being tested statistically.
Different α-levels have different advantages and disadvantages. Smaller α-levels give greater confidence in the determination of significance, but run greater risks of failing to reject a false null hypothesis (a
Type II error, or "false negative determination"), and so have less
statistical power. The selection of an α-level inevitably involves a compromise between significance and power, and consequently between the
Type I error and the
Type II error.
In some fields, for example nuclear and particle physics, it's common to express statistical significance in units of "σ" (sigma), the
standard deviation of a
Gaussian distribution. A statistical significance of "
" can be converted into a value of α via use of the
error function:
»
For clarity, the above formula is presented in tabular form below.
Dependence of confidence with noise, signal and sample size (tabular form)
| Parameter |
Parameter increases |
Parameter decreases |
| Noise |
Confidence decreases |
Confidence increases |
| Signal |
Confidence increases |
Confidence decreases |
| Sample size |
Confidence increases |
Confidence decreases |
In words, the dependence of confidence is high if the noise is low and/or the sample size is large and/or the
effect size (signal) is large. The confidence of a result (and its associated
confidence interval) is
not dependent on effect size alone. If the sample size is large and the noise is low a small effect size can be measured with great confidence. Whether a small effect size is considered important is dependent on the context of the events compared.
In medicine, small effect sizes (reflected by small increases of risk) are often considered clinically relevant and are frequently used to guide treatment decisions (if there's great confidence in them). Whether a given treatment is considered a worthy endeavour is dependent on the risks, benefits and costs.
Further Information
Get more info on 'Statistical Significance'.
|
External Link Exchanges
Do you know how hard it is to get a link from a large encyclopaedia? Well we're different and will prove it. To get a link from us just add the following HTML to your site on a relevant page:
<a href="http://statistical_significance.totallyexplained.com">Statistical significance Totally Explained</a>
Then simply click through this link from your web page. Our crawlers will verify your link, extract the title of your web page and instantly add a link back to it. If you like you can remove the words Totally Explained and embed the link in article text.
As long as your link remains in place, we'll keep our link to you right here. Please play fair - our crawlers are watching. Your site must be closely related to this one's topic. Any kind of spamming, dubious practises or removing the link will result in your link from us being dropped and, potentially, your whole site being banned. |
